Temporal Symbolic Integration Applied to a Multimodal System Using Gestures and Speech
نویسندگان
چکیده
This paper presents a technical approach for temporal symbol integration aimed to be generally applicable in unimodal and multimodal user interfaces. It draws its strength from symbolic data representation and an underlying rulebased system, and is embedded in a multi-agent system. The core method for temporal integration is motivated by findings from cognitive science research. We discuss its application for a gesture recognition task and speech-gesture integration in a Virtual Construction scenario. Finally an outlook of an empirical evaluation is given.
منابع مشابه
Differential roles for left inferior frontal and superior temporal cortex in multimodal integration of action and language
Several studies indicate that both posterior superior temporal sulcus/middle temporal gyrus (pSTS/MTG) and left inferior frontal gyrus (LIFG) are involved in integrating information from different modalities. Here we investigated the respective roles of these two areas in integration of action and language information. We exploited the fact that the semantic relationship between language and di...
متن کاملUsing Contextual Knowledge to Inform a Multimodal Interactive System
An essential feature for personal robots in domestic environments is the quality of its interaction. This work describes a Contextually Informed MultiModal Integrator, CIMMI, that fuses speech and symbolic gestures probabilistically and is informed by contextual knowledge in an assistive technology robotic application. Symbolic gestures are gestures that have semantic meaning, such as a wave ge...
متن کاملIntegration of iconic gestures and speech in left superior temporal areas boosts speech comprehension under adverse listening conditions
Iconic gestures are spontaneous hand movements that illustrate certain contents of speech and, as such, are an important part of face-to-face communication. This experiment targets the brain bases of how iconic gestures and speech are integrated during comprehension. Areas of integration were identified on the basis of two classic properties of multimodal integration, bimodal enhancement and in...
متن کاملToward Natural Gesture/Speech Control of a Large Display
In recent years because of the advances in computer vision research, free hand gestures have been explored as means of human-computer interaction (HCI). Together with improved speech processing technology it is an important step toward natural multimodal HCI. However, inclusion of non-predefined continuous gestures into a multimodal framework is a challenging problem. In this paper, we propose ...
متن کاملHPSG Approach to Synchronous Speech and Deixis
The use of hand gestures to point at objects and individuals, or to navigate through landmarks on a virtually created map is ubiquitous in faceto-face conversation. We take this observation as a starting point, and we demonstrate that deictic gestures can be analysed on a par with speech by using standard methods from constraint-based grammars such as HPSG. In particular, we use the form of the...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 1999